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Abstract 


Focusing on the science knowledge and pedagogical content knowledge that teachers need to 
realize the vision of the Next Generation Science Standards (NGSS) in their classrooms, this 
study presents findings from efforts to (1) adapt existing criteria and design heuristics for 
educative features of curriculum materials to the needs of NGSS and (2) develop an authentic 
measure of teachers’ knowledge and practice based on an analysis of teachers’ evaluations of 
their students’ written explanations of phenomena. The study demonstrates that existing criteria 
and heuristics for designing educative features of curriculum materials can be used productively, 
with minor modifications, to design features that support teachers in their use of materials that 
support NGSS. It also provides quantitative and qualitative data to show that teachers’ analyses 
of the explanation task produced useful information on their understanding of the science ideas 
targeted in the assessment, of the misconceptions their students held, and of their students’ 
ability to reason from evidence, science ideas, and models in explaining phenomena. This study 
builds on and contributes to a body of work on the design and use of educative curriculum 
materials and the evaluation of teacher knowledge. It suggests a practical approach to the design 
of NGSS-aligned curriculum materials that support both student and teacher learning based on 


findings from analysis and empirical studies. 
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Designing for the Next Generation Science Standards: Educative Curriculum Materials and 
Measures of Teacher Knowledge 

With the release of Next Generation Science Standards (NGSS) (NGSS Lead States, 
2013) and its adoption by 18 states and the District of Columbia, science educators are being 
asked to transform the way they think about the content they teach and the way that they teach it. 
Among other changes, NGSS—along with the National Research Council’s [NRC] 4 
Framework for K-12 Science Education (2012) that preceded it—emphasizes the interconnected 
nature of science as it is practiced and experienced in the real world, a coherent progression of 
science concepts from kindergarten through high school, and a deeper understanding of core 
science ideas and their usefulness in making sense of phenomena and developing solutions to 
problems. 

To implement NGSS, teachers must reconsider the science content that is taught, how 
students’ build their understanding of that content, and how ideas fit together to tell a coherent 
story (Reiser, 2013). While professional development (PD) can provide some help to teachers for 
these tasks, limited time and other resources make PD, by itself, an impractical solution to satisfy 
the needs of teachers across all grades and science disciplines. According to Achieve, a partner 
organization in the development of NGSS, appropriate instructional materials designed to 
support NGSS will have an important role to play. To help teachers understand the new 
standards and implement them effectively, says Achieve, materials will need to provide an 
extensive range of supports, from suggestions for how to engage students in developing 
explanations and constructing conceptual models of the natural world to learning progressions 
that map out students’ development of science content knowledge over time (2015). Indeed, 


because of their widespread use by teachers (Horizon Research, 2012), curriculum materials 


have enormous potential for leveraging teachers’ role as enactors of curriculum to transfer of 
research findings, best practices, and reforms such as those proposed by NGSS into the 
classroom (Ball & Cohen, 1996; Remillard, 2005). 

More than a decade ago, Davis and Krajcik (2005) broke new ground by proposing a set 
of heuristics for designing educative science materials that would support teachers in developing 
their (1) subject matter knowledge (SMK); (2) pedagogical content knowledge (PCK) for science 
ideas; and (3) PCK for scientific inquiry, i.e., their ability to engage and guide students in using 
science practices. Their work focused on the kinds of knowledge that teachers would need to 
meet the practical challenges of the classroom. Today, in the era of NGSS, teachers face new 
challenges and need examples of what effective instruction and curriculum might look like 
(National Research Council, 2015). 

One such example, we believe, is a middle school curriculum unit that is designed to 
promote student achievement of the NGSS vision while also providing teachers with the 
educative features they need to implement that vision in their classrooms. Developed by a team 
of researchers and curriculum developers at AAAS and BSCS, the Toward High School Biology 
(THSB) unit is one of the first materials intentionally designed to realize the NGSS vision of 
three-dimensional learning. 

To date, there has been little exploration of the role that educative curriculum materials 
can play in promoting the NGSS vision of science learning to a broad audience of teachers. In 
addition, the research literature on educative materials consistently points to difficulties that 
many investigators have had in developing appropriate measures for evaluating teacher 
knowledge and, hence, the effects of educative materials on teachers’ knowledge and practice. 


This paper aims to help fill that gap by providing evidence from the development and testing of 


the THSB unit to answer the following questions: (1) Can the design heuristics proposed by 
Davis and Krajcik (2005) be adapted and used to inform the design of educative materials for 
NGSS? (2) Can the same heuristics also guide the development of authentic measures to evaluate 
teacher knowledge and practice? 

Theoretical Foundation for Educative Curriculum Materials 

Design of the THSB unit as a whole was informed by the general view that curriculum 
materials can have a powerful impact on what and how science is taught for the benefit of 
students and teachers alike (Kali, Koppal, Linn, & Roseman, 2008). More specifically, the 
developers applied curriculum design principles that emerged from research on the coherence, 
quality, and effectiveness of middle and high school science textbooks (Herrmann-Abell, 
Koppal, & Roseman, 2016; Kesidou & Roseman, 2002; Roseman, Stern, & Koppal, 2010; Stern 
& Roseman, 2004) and from the learning research that underpins these principles (Bransford, 
Brown, & Cocking, 2000; Clements, 2007; NRC, 2007; Pashler et al., 2007). 

With the release of the Framework and NGSS in 2012 and 2013, the THSB unit was 
revised to meet the criteria for measuring the alignment and quality of individual lessons and 
units with respect to the NGSS found in the Educators Evaluating the Quality of Instructional 
Products (EQuIP) NGSS Rubric (Achieve, 2014; Roseman, Fortus, Krajcik, & Reiser, 2015). 
Design of the educative features of the THSB unit drew on work exploring the potential of 
curriculum materials to serve as a source of teacher learning (Ball & Cohen, 1996; Davis & 
Krajcik, 2005; Schneider, Krajcik, & Marx, 2000). 

Pre-Service Teachers’ Use of Educative Curriculum Materials 
Given the novice level of pre-service teachers, their use of curriculum materials is of 


particular interest to researchers exploring the role that educative materials might play in 


preparing teachers for the classroom. For example, Beyer and Davis (2009) examined the use of 
educative curriculum materials in improving the ability of pre-service elementary teachers to 
critique and adapt science curriculum materials. They focused on educative features designed to 
support teachers in applying the principles of identifying, interpreting, and working with 
students’ science ideas. Pre- and post-tests found that teachers’ use of the principles in their 
analyses was higher when the supports were present compared to when the supports were absent. 

But simply providing educative features in materials may not be sufficient for teacher 
learning. A 2015 study by Land, Tyminski, and Drake found that even when educative features 
were present in the materials, the pre-service mathematics teachers tended not to read them in 
educative ways (e.g., their interpretations of the materials were more descriptive than analytic, 
highlighting aspects of lessons that were “fun’’). Confirming findings from their earlier study 
(Drake, Land, & Tyminski, 2014) and consistent with the work of Beyer and Davis (2009), Nicol 
and Crespo (2006), and Schwarz et al. (2009), the authors concluded that pre-service teachers 
need help in making sense of educative curriculum materials and require help in using 
frameworks for analyzing materials productively. 
In-Service Teachers’ Use of Educative Curriculum Materials 

Of particular relevance to the implementation of NGSS is the role that educative 
curriculum materials can play in helping teachers who are already in the classroom make 
changes in their instruction that will support their students’ achievement of NGSS performance 
expectations. Findings have shown that in-service teachers do benefit from the educative 
features, but not always and not always as expected (McNeill, 2008; McNeill & Krajcik, 2009; 
Schneider & Krajcik, 2002). Most of the studies were small in size, largely descriptive in nature, 


and lacked reliable measures of what teachers learned. In the only study dealing directly with 


NGSS, Bismack, Arias, Davis, and Palinscar (2015) enhanced a 4th grade unit to increase 
support for three science practices (recording observations, making scientific predictions, and 
making evidence-based claims). Findings indicated that the educative features improved 
students’ observations and predictions but not their evidence-based claims. Teachers’ comments 
suggested that they were “uncertain how to make and engage students in this practice,” and the 
authors highlighted the need for more extensive research on the design of educative features to 
support this and other science practices. This study focused explicitly on the science practices 
dimension in isolation rather than integrating it with the other two NGSS dimensions of core 
disciplinary ideas and crosscutting concepts. 

Measures of Teacher Knowledge 

The field has found it difficult to develop both valid and authentic measures that 
adequately capture all that teachers must know and be able to do in the classroom. At one end of 
the spectrum, for example, Garet, Porter, Desimone, Birman, and Yoon (2001) used self-reports 
on a Likert scale survey to measure teachers’ knowledge and skills, and Rowan, Correnti, and 
Miller (2002) used items from large-scale surveys to measure content knowledge and PCK in 
mathematics and writing. They found the survey more successful in measuring such knowledge 
in the context of mathematics than writing and more successful in measuring mathematics 
content knowledge than mathematics PCK. 

At the other end of the spectrum, researchers have used more authentic tasks to measure 
teacher knowledge, including evaluating teachers’ ability to analyze videotapes of teaching or to 
analyze curriculum materials or other artifacts of instruction. For example, Kersting (2008) used 
teachers’ ability to analyze videotapes of mathematics teaching as a proxy for their teaching 


knowledge, and Santagata, Zannoni, and Stigler (2007) used a video-based program on lesson 


analysis in a pre-service program for mathematics teachers. They measured pre-service teachers’ 
ability to analyze a new lesson using the lesson analysis framework taught in the program. 
Schwarz, Gunckel, Smith, Covitt, and Bae (2008) explored the use of curriculum analysis— 
specifically Project 2061’s curriculum materials analysis criteria (AAAS, 2005)—1in pre-service 
elementary teacher education. Teachers were given six activities about light and shadows and 
asked to (a) assess strengths and weaknesses of each, (b) write a lesson sequence that briefly 
describes the activities they would use/adapt, and (c) provide a rationale for their sequence. Pre- 
and post-test results showed an increase in the use of criteria related to content alignment and 
attention to some of the instructional criteria, but the study also found that teachers had their own 
criteria (e.g. making science fun) and were generally negative about the use of the Project 2061 
criteria. 

Other studies have used teachers’ ability to attend to their students’ thinking as a measure 
of their knowledge. Jacobs, Lamb, and Phillipp (2010) developed a framework that included (a) 
attending to children’s strategies, (b) interpreting children’s mathematical understandings, and 
(c) deciding how to respond on the basis of children’s understanding. Teacher knowledge was 
measured by having teachers observe a video excerpt of an elementary mathematics classroom 
and prepare a written report that responded to prompts for the three categories in the framework. 

Overview of the THSB Unit 

Developed by a team of scientists, education researchers, and curriculum developers at 
AAAS and BSCS, the THSB curriculum intervention is an eight-week replacement unit for 8th 
grade science classes. It includes (1) a print student edition (SE) workbook, (2) a print teacher 
edition (TE), (3) web-based teacher resources, and (4) a three-day introductory face-to-face PD 


workshop. The team used an iterative design process that involved the initial conceptualization 


and writing of the unit; multiple rounds of testing the unit in classrooms; and multiple revisions 
of the unit based on student performance data, curriculum analysis, and feedback from teachers. 
Over the course of the five-year development project, more than 1000 students and 12 teachers 

were involved in various phases of the design and testing of the curriculum intervention. 

The overarching goal of the THSB unit is to help students make sense of phenomena 
related to plant and animal growth using NGSS core ideas about chemical reactions in nonliving 
and living systems; the crosscutting concept of matter conservation across physical and life 
science; and science practices of data analysis, modeling, explanation, and communication. 

The THSB unit consists of 19 lessons organized into four chapters. Chapter 1 develops 
the idea that substances with different properties form during chemical reactions because the 
atoms that make up molecules of the starting substances rearrange to form the molecules of the 
ending substances. Chapter 2 develops the idea that the total mass stays the same during 
chemical reactions because the number of each type of atom stays the same and the mass of each 
atom stays the same regardless of differences in their arrangement. Chapter 3 applies ideas of 
atom rearrangement and conservation to plant growth and repair, and Chapter 4 applies these 
ideas to animal growth and repair. Table 1 shows how the phenomena that students experience in 
each chapter relate to the chapter’s conceptual focus. 

[insert Table | here] 
Research Question 1: Can the design heuristics proposed by Davis and 
Krajcik (2005) be adapted and used to inform the design of educative materials for NGSS? 
Methods 
Elsewhere, we and our colleagues have described initial efforts of the THSB 


development team to design and test the promise of educative materials in supporting teachers in 


their use of the THSB unit (Flanagan, Herrmann-Abell, & Roseman, 2013; Kruse, Howes, 
Carlson, Roth, & Bourdelat-Parks, 2013). The team drew on theories of teacher learning and the 
role of educative materials articulated by Ball and Cohen (1996) and Davis and Krajcik (2005) 
and on AAAS’s own criteria for evaluating the quality of curriculum materials, including the 
support that materials provided for teachers (2005). The THSB development team distilled from 
these sources an initial set of principles that was THSB-specific and focused on the design of 
supports to help teachers develop their (1) knowledge of the content and practices that make up 
THSB’s storyline at the unit, chapter, and lesson levels; (2) knowledge of the pedagogical 
purposes of each component of the unit (e.g., each phenomenon, model, and activity); (3) 
knowledge of teaching strategies that help to make the unit storyline visible to students and that 
help teachers elicit, listen to, and respond to students’ ideas and questions; and (4) ability to take 
an analytical approach toward planning and implementing the unit and monitoring their students’ 
progress (Kruse et al., 2013). All four components of the THSB unit—the SE, TE, online 
resources, and PD—were designed to contribute in different ways to teachers’ understanding of 
and skill in using the unit. For the purposes of the study reported on here, however, we focus 
mainly on the educative features of the SE, TE, and online resources. 

Following the release of the NRC Framework and NGSS in 2012 and 2013, the 
development team improved the unit’s alignment with the three-dimensional vision of the new 
standards and considered what additional supports teachers would need to understand and 
implement it. In doing so, the team focused on aspects of NGSS that were central to the THSB 
unit, doable within eight weeks of instructional time, and that would illustrate what it means to 


use ideas and practices to make sense of phenomena. 
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To take into account the implications of NGSS for the design of the THSB unit’s 
educative features, the development team first identified critical aspects of NGSS that were not 
addressed explicitly in the design heuristics proposed by Davis and Krajcik but were essential for 
teachers to understand in order to help their students achieve the goals of NGSS. Given the broad 
scope of the changes called for in NGSS and the busy lives of teachers, the development team 
tried first to ensure that the “’base’ curriculum materials were accurate, complete, and coherent 
in terms of content and effective in terms of pedagogy—with good representations of the 
content, a clear purpose for learning it, and multiple opportunities for students to explain their 
ideas” (Davis & Krajcik, 2005, p. 3). In doing so, the team confronted what Davis and Krajcik 
refer to as “tensions in determining an appropriate amount of guidance and prescription” (2005, 
p. 9). Our own observations during pilot testing had indicated that several teachers used only the 
SE while teaching, so we opted to incorporate a good deal of instructional support for integrating 
core disciplinary ideas, science practices, and crosscutting concepts directly into the SE. 
Additional educative features—particularly support for science practices—were then developed 
and used by teachers during field tests of the unit and revised based on their feedback during de- 
briefing sessions. 

Adapting the Design Heuristic for SMK 

Davis and Krajcik’s design heuristic for SMK calls for materials to, among other things, 
support teachers in “developing factual and conceptual knowledge of science content.... at a 
level beyond the level of understanding required by the students” (2005, p. 12). It also calls for 
materials to help teachers see how the “scientific ideas relate to real-world phenomena and to the 
activities in the unit....”” We interpreted this heuristic to include knowledge of NGSS disciplinary 


core ideas and crosscutting concepts and how they are organized into a coherent story, which is 


11 


consistent with best practices gleaned from an analysis of science lessons from five countries in 
the TIMSS Video Study (Roth et al, 2006). To design specific lessons and activities for a 
curriculum unit situated within a specific context (i.e., developing a molecular explanation for 
growth and repair in living things), it was necessary to extract from the disciplinary core ideas 
and crosscutting concepts a set of contextualized “science ideas” that would provide a coherent 
content storyline for the THSB unit. 

Organizing SMK around a coherent content storyline is critical for both students and 
teachers, so that both come to see the big picture that encompasses the science ideas and the 
contribution of each activity to its development. Figure 1 presents a “map” of the THSB science 
ideas showing how they were sequenced, starting with disciplinary core ideas and culminating in 
ideas at the top of the map that draw on both disciplinary core ideas and crosscutting concepts. 
Also included in this SMK heuristic is information on the contribution of each lesson to the 
development of the content storyline. Each numbered box in Figure | identifies not only the 
relevant NGSS code but also the THSB lesson(s) in which the science idea is targeted. 

[insert Figure | here] 

In designing curriculum for NGSS, it was also important to consider what teachers would 
need to know about the role of crosscutting concepts (in this case the idea that atom conservation 
explains mass conservation) as science ideas themselves and as a distinct dimension of science 
learning that serves to unite core ideas across the disciplines to explain phenomena and answer 
questions. (NGSS Lead States, 2013). Although the power of certain ideas that cut across 
disciplines was recognized in Science for All Americans, which describes such ideas as “tools for 
thinking about how the world works” (AAAS, 1989, p. 19), and in the NRC Framework, which 


describes crosscutting concepts as providing “an organizational framework for connecting 
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knowledge” (2012, p. 83), neither provides guidance about how connections should be made or 
how to support teachers in guiding their students to make them. In developing the THSB unit, we 
interpreted crosscutting concepts as tools for thinking across a wider range of phenomena than 
could be explained with particular core ideas. Additional supports would be needed to help 
teachers appreciate this explanatory value of the crosscutting concepts. 

Adapting Design Heuristics for PCK for Science Topics 

The heuristics proposed by Davis and Krajcik for this category focus on supporting 
teachers in engaging their students with phenomena, in using scientific representations, and in 
addressing students’ ideas, all of which are relevant to realizing the three-dimensional vision of 
NGSS. 

Phenomena. Making sense of phenomena is at the heart of NGSS, and students are 
expected to engage in science practices and use disciplinary core ideas and crosscutting concepts 
to explain them. NGSS also emphasizes the interconnectedness of science with the goal of 
helping students see how a set of core ideas can be used to explain different phenomena across 
multiple science disciplines (NRC, 2012). With this in mind, we elaborated on the Davis and 
Krajcik design heuristics to include educative features that provide teachers with a rationale for 
the inclusion of the chosen phenomena along with support in carrying out activities designed to 
help students see the explanatory power of science ideas, how they fit together, and how they can 
apply to a range of physical and life science phenomena. 

To capture the integrated learning across disciplines that is called for in NGSS, we 
distinguished two categories of phenomena: those that have the potential to be generative of the 
targeted science ideas and those that are not likely to be generative of the ideas but could be 


explained by them once the ideas were understood. For example, physical science phenomena 
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that can be carried out with pure substances are more likely to be generative of the ideas that new 
substances can be produced from starting substances (which can be directly observed) and that 
the molecules making up the new substances can be made by rearranging the atoms making up 
the molecules of starting substances (which can be modeled). The need for phenomena to be 
generative was considered in our decision to include chemical reactions in nonliving and living 
systems in the same unit and to start with chemical reactions in nonliving systems, and we 
wanted teachers to understand the rationale for selecting and sequencing these phenomena. 

Representations. This Davis and Krajcik design heuristic focuses on helping teachers 
select and adapt appropriate representations for their students and identifying features of a 
representation that are the most salient for instruction. Educative curriculum materials can also 
support teachers in becoming aware of and making use of a wide variety of representations— 
illustrations, tables and graphs, diagrams, physical models, and simulations—that can help to 
make phenomena accessible to a wide range of students (Kesidou & Roseman, 2002; Roseman, 
Stern, & Koppal, 2010). We found support for this adaptation of the heuristic in the EQuIP 
Rubric (OR 1), which recognizes the role that representations, among other things, can play in 
NGSS-focused materials as tools for differentiating instruction for students who struggle as well 
as for those who are more advanced (Achieve, 2016). 

Students’ ideas. Educative curriculum materials can help teachers understand, identify, 
and deal with intuitive and non-normative science ideas that many of their students bring to the 
classroom (Driver, Squires, Rushworth, & Wood-Robinson, 1994). Indeed, teacher’s awareness 
of students’ misconceptions has been shown to correlate to student learning gains (Sadler, 
Sonnert, Coyle, Cook-Smith, & Miller, 2013). Studies have shown that teachers typically do not 


focus on their students’ ideas (Sherin & van Es, 2005), so materials and accompanying PD need 
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to help them do so. Again, we drew on the EQuIP Rubric (OR 1) for NGSS-specific guidance. 
The Rubric has eight criteria in three categories that include the need for instructional supports to 
help teachers identify students’ ideas and for assessment rubrics and scoring guidelines to help 
teachers plan instruction and give feedback to their students. 
Adapting Design Heuristics for PCK for Scientific Inquiry 

The NRC Framework and NGSS clarify scientific inquiry with a set of eight science 
practices that relate closely to the design heuristics for PCK for scientific inquiry in Davis and 
Krajcik (2005). Though listed separately, NGSS makes clear that the practices work together to 
help students make sense of phenomena and that reflecting on their use can help students 
understand how the practices contribute to the development of scientific knowledge. The THSB 
unit focuses intensely on three science practices—data analysis, modeling, and explanation. Each 
is developed in the context of making sense of specific phenomena in nonliving and living 
systems in terms of ideas about atom rearrangement and conservation during chemical reactions. 

In adapting this design heuristic for use with materials that support NGSS, we tried to 
address the knowledge that teachers would need in order to help foster their students’ integrated 
science learning across the three dimensions of disciplinary core ideas, science practices, and 
crosscutting concepts. Alignment with the NGSS vision would, at the least, require that (a) 
students engage in explaining phenomena; (b) students’ explanations would drive their learning; 
and (c) their explanations would require the use of all three dimensions—disciplinary core ideas, 
science practices, and crosscutting concepts. To achieve this integration, teachers would need 
activities that are designed explicitly for this purpose and additional supports to help them see 
how the three dimensions work together to deepen students’ understanding. 


Using the EQuIP Rubric to Guide Three-Dimensional Design 
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In 2014 the first version of the EQuIP Rubric was released by Achieve to provide criteria 
for analyzing the extent to which lessons and units are designed for the NGSS (Achieve, 2014). 
The development team used these criteria and the revised versions of the criteria released in 
subsequent years (Achieve, 2016) as an additional check on the extent to which the THSB unit’s 
educative features were likely to support teachers in achieving NGSS’s vision of three- 
dimensional learning. 
Results 
Based on the adapted heuristics described above, we designed the THSB unit itself, along 

with specific features just for teachers, to educate teachers about the unit and how to implement 
it in their classrooms. The design of the unit itself was intended to build teachers’ awareness of 
NGSS’s vision for three-dimensional learning and how the THSB unit supports that vision. 
Educative Features in THSB to Promote Teachers’ SMK 

THSB teacher materials include several components to clarify the ideas targeted, how they 
are organized into a coherent content storyline, and the contribution of specific lessons and 
activities to student learning. The TE unit introduction lists the specific NGSS disciplinary core 
ideas, crosscutting concepts, and science practices that are targeted (see pp. i1i-v in OR 2), shows 
how they are organized into a coherent storyline (p. viii in OR 2), and provides a rationale for the 
inclusion and exclusion of core ideas, crosscutting concepts, and vocabulary (see pp. vii-xi in OR 
2), 

To deepen teachers’ understanding of the relevant content, each chapter in the TE includes 
a Background Knowledge for Teachers section that provides essential information about the 
science ideas targeted in the upcoming lessons. The Chapter Overviews (see OR 2 for an 


example) and Lesson Guides (see OR 4 for an example) describe the contribution of each lesson 
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and activity to the progression of the content storyline, including how the crosscutting concept is 
developed in each chapter. 
Educative Features in THSB to Promote Teachers’ PCK for Science Topics 
Phenomena. Chapters | and 2 of the THSB SE start with a set of specific phenomena 
that are generative of each of the science ideas targeted, and the TE helps teachers understand the 
role that each phenomenon plays in advancing the content storyline. The Unit Overview 
describes how observations and data are used to provide evidence for the science ideas and/or 
their explanatory power (see p. vi in OR 2), the Chapter Overview describes the rationale for the 
specific phenomena included (see pp. 1b-1d in OR 3), and each Lesson Overview describes the 
purpose of each activity and the intended observations students should make, either through 
direct observations or inferences from data (see pp.44c-d in OR 4 and p. 60c in OR 5). 
Representations. The THSB unit uses a variety of representations to make atoms, 


molecules, and chemical reactions concrete to students, including conventional models, e.g., 


space-filling and ball-and-stick models, and unconventional models, e.g., LEGO® bricks and 


flattened two-dimensional versions of ball-and-stick models. Chemical reactions are represented 
by chemical reaction mats, as shown in Figure 2, which display molecular models of starting and 
ending substances. 

[insert Figure 2 here] 
Modeling activities help students see that it is possible to build the product molecules just by 
rearranging the atoms of the reactant molecules. They also see how it is possible for measured 
mass to increase even though total mass remains the same, an observation they will use when 
explaining why the growth of mushrooms on a fallen dead tree does not violate conservation 


principles. 
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The TE Unit Overview (pp. vi— vii in OR 2) supports teachers in understanding the role 
of models in the THSB unit and the intended observations students should make when using 
them. To help ensure the modeling activities are used to explain the phenomena in terms of the 
science ideas, the Lesson Guides describe the intended observations students should make, as 
shown in Figure 3 from a lesson targeting ideas about conservation of mass and measured mass. 

[insert Figure 3 here] 

Students’ ideas. The THSB unit provides a variety of opportunities for students to 
express their initial ideas, engage in activities that build on or challenge them, contrast their ideas 
to scientific ideas, and consider how their ideas have changed after experiencing the activities. 
For example, after interpreting data from radioactive labeling experiments that provide evidence 
that plants use the carbon and oxygen atoms from carbon dioxide to make glucose and use 
glucose to make cellulose, students examine Van Helmont’s willow tree data, critique his 
(invalid) conclusion that most of the increased mass comes from water, and then construct a valid 
explanation based on the data they have examined (see pp. 136-138 in OR 6). 

The THSB unit supports teachers in identifying, building on, challenging, and monitoring 
students’ ideas throughout the unit. Every TE Chapter Overview alerts teachers to commonly 
held student ideas, their manifestations in student work, and the role of specific activities in 
addressing them (see pp. 1d-le in OR 3). Additionally, the THSB includes embedded assessment 
tasks at the end of each chapter (see Table 2) and online resources provide rubrics and scoring 
guidelines for each task along with examples of how misconceptions might be expressed in 
student explanations (see pp. 2-3 and 5-6 in OR 9). 

[insert Table 2 here] 


Educative Features in THSB to Promote Teachers’ PCK for Scientific Inquiry 
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Analyzing and interpreting data. The THSB unit provides many opportunities for 
students to collect, analyze, and interpret data. Students determine properties of substances 
through direct observation (e.g., color, state of matter at room temperature, electrical 
conductivity, water solubility, ability to form fibers, behavior in splint and limewater tests), 
identify patterns in their observations (e.g., correlations between increasing amounts of ending 
substances with decreasing amounts of starting substances), and supplement direct observations 
with information in data tables from published scientific studies (e.g., data from radioactive 
labeling experiments). Students use their observations and the data as evidence that new 
substances were produced, that the ending substances were actually produced from the starting 
substances, and that atoms making up the molecules of starting substances were incorporated 
into the molecules of the ending substances (e.g., algae incorporate '4C atoms from carbon 
dioxide into glucose, cress plants incorporate '4C atoms from glucose into cellulose [see OR 8], 
young herring fish incorporate '*C atoms from brine shrimp proteins into their own body 
proteins). Carefully sequenced questions guide students in interpreting data and in using their 
interpretations as evidence to support claims. The TE provides ideal student responses to those 
questions (see p. 119 of OR 8) and includes in each Lesson Guide a chart summarizing activities 
in which students analyze and interpret data to provide evidence for phenomena; clarifying the 
purpose of each activity; and listing intended observations, inferences from data, and conclusions 
that students should make (see Figure 3). The Closure notes for teachers at the end of each lesson 
provide the main points that should emerge from a class discussion about the data analysis tasks 
(see p.125 of OR 8). 

Modeling. The THSB unit engages students in modeling activities that help them make 


sense of observations and data, namely that molecules making up the ending substances can be 
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made by rearranging the atoms making up molecules of the starting substances. For example, 
students use ball-and-stick models to show how a fibrous solid (made up of nylon polymers) 
might be made from two clear colorless liquids (each made up of a single monomer) and how 
algae might incorporate '4C atoms from carbon dioxide into glucose, cress plants might 
incorporate '*C atoms from glucose into cellulose, and young herring fish might incorporate '4C 
atoms from brine shrimp proteins into their own body proteins. As with data analysis, carefully 
sequenced questions guide students in observing and interpreting their modeling activities. 

The teacher resources support teachers in helping students carry out the modeling 
activities and observe, interpret, and draw the intended conclusions. Online how-to videos of the 
more complicated modeling activities help teachers become more familiar with handling the 
models and provide examples of how the models should be used during the activities. The TE 
includes notes on how to encourage students to use models to make sense of novel phenomena 
and how students should use ideas from modeling activities to support their explanations. 

Constructing explanations. The THSB unit engages students in constructing scientific 
explanations of phenomena, which is one of the key scientific practices in NGSS. Because of the 
complexity of this practice and the difficulties students have with it, THSB provides support 
consistent with the cognitive apprenticeship model (Brown, Collins, & Duguid, 1989; Collins, 
Brown, & Newman, 1989). Early in the unit, students are provided with an example of a valid 
explanation, and the essential elements of an explanation—claim, relevant science ideas, and 
evidence that will be linked to the claim using science ideas—are defined. The early lessons also 
establish Explanation Quality Criteria that students can use in judging an explanation’s quality 
and introduce a template to help them organize their thinking and writing around the essential 


elements. After students experience modeling activities, a new element is added to the 
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explanation template—models and modeling as tools for thinking about how something might 
work—and from then on students are expected to include models in their reasoning (see p. 51 in 
OR 4). As the unit proceeds, students are reminded to use ideas about atoms in their explanations 
and to be sure that their explanations meet the unit’s Explanation Quality Criteria. The 
explanation template is faded in the SE but is available in the TE if the teacher feels students still 
need it. At the end of the unit, the scaffolding is almost completely removed with the exception 
of a reminder that students can use science ideas from anywhere in the unit in their explanation. 

To support teachers in helping students construct evidence-based explanations, the TE 
provides ideal responses for each explanation task and for each element of the explanation 
template. The TE also provides discussion notes (see p. 58a in OR 4) to help teachers guide their 
students in using the science ideas and considering the role that models play in their 
explanations. In addition, as described below, online teacher resources provide rubrics for 
scoring the explanation tasks that are recommended for use as embedded assessments. 
Educative Features to Promote Three-Dimensional Teaching and Learning 

Table 3 presents data from an analysis of the unit using version 3.0 of the EQuIP Rubric 
criteria to demonstrate how the THSB unit meets the criteria for integrating the three dimensions 
of NGSS (Category | criteria), supporting three-dimensional teaching and learning (Category 2 
criteria), and monitoring student progress in all three dimensions (Category 3 criteria). 

[insert Table 3 here] 

In addition to the evidence cited in Table 3, the THSB unit provides many other 
opportunities for students and teachers to achieve the NGSS goal of integrating core disciplinary 
ideas, science practices, and crosscutting concepts to explain a range of phenomena in physical 


and life science settings. For example, Chapter 2 science ideas about atom conservation in 
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chemical reactions, modeling and explanation practices, and the crosscutting concept of atom 
conservation explaining mass conservation help students make sense of observations of changes 
in measured mass in open containers such as when baking soda reacts with vinegar, iron reacts 
with oxygen, a log burns in a fireplace, and copper in the Statue of Liberty reacts with substances 
in the atmosphere. 

Similarly, in Chapters 3 and 4, students develop explanations of plant and animal growth, 
both of which involve increases in measured mass, by making sense of data about mass increases 
in a growing willow tree, the incorporation of '*C from glucose into cellulose in an experiment 
that tests the effect of an herbicide on growth of cress plants, and the incorporation of '4C from 
brine shrimp proteins into herring fish body proteins in an experiment investigating possible food 
sources for fish farming. Lesson Guides in the TE identify the ideas and practices that are 
integrated in each lesson (see p. 60b in OR 5). 

Embedded assessment tasks at the end of each chapter tap all three dimensions of science 
learning and serve as examples of what it means to integrate those dimensions in teaching and 
learning. The assessment tasks require students to use (a) science ideas, including those derived 
from both disciplinary core ideas and crosscutting concepts (see the content storyline map in 
Figure 1); (b) evidence gathered by students as they engage in the practices of observation and 
data analysis; and (c) reasoning from models and modeling activities to explain a phenomenon. 
For example, in one task students are asked to explain why the growth of mushrooms on a fallen 
dead tree does not violate the concept of matter conservation. Suggested answers and rubrics for 
scoring tasks such as these help teachers see how disciplinary core ideas, practices, and 


crosscutting concepts can be integrated in assessments (see OR 9). 
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Face-to-face PD extended the discussion of three-dimensional learning in the THSB unit 
in several ways. In sessions for experienced THSB teachers, for example, teachers (a) examined 
“evidence tables” that summarize how the unit integrates each science idea with each practice 
(see OR 10); (b) considered the range of phenomena across the unit’s four chapters that students 
observe, model, and explain; and (c) used the rubrics to score samples of student work. 

Having a set of appropriate criteria for designing educative features and ensuring that the 
materials meet those criteria are essential first steps in increasing the knowledge and skills 
teachers need to implement the NGSS vision. It is also essential to find out whether the educative 
features actually do increase teachers’ knowledge and skills. This requires valid measures of 
teacher knowledge and practice that could be used in future studies of what teachers learn from 
educative curriculum materials. 

Research Question 2: Can the same design heuristics also guide the development of 
measures to evaluate teacher knowledge and practice? 

The first iteration of a measure was an online test of teachers’ knowledge of science 
ideas, relevant student misconceptions, and phenomena that was administered before PD, after 
PD, and after teaching the unit (Flanagan, et al., 2013). While the data from the teacher test was 
useful in informing revisions to the unit, the variability in teacher performance across contexts 
indicated that the assessment was not yet a valid measure of teacher knowledge. We also wanted 
to measure teachers’ knowledge using a more authentic task—one that they would view as being 
useful to their work. We focused on using teachers’ analysis and synthesis of their students’ 
responses to the unit’s embedded assessments as a possible measure of teacher knowledge. 

Methods 


Participants 
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During the 2015 implementation of the unit, six middle school teachers from a mid- 
Atlantic suburban school district participated in a study involving the embedded student 
assessments included at the end of each chapter in the THSB unit (see Table 2). Table 4 
provides information about the participating teachers, including their prior experience with the 
THSB unit. Teachers had participated in three days of PD prior to their first year using the unit. 

Insert Table 4 here 

The teachers participated in an additional one-day of PD that updated them on revisions 
to the unit and why they were made and provided them with an opportunity to examine and 
discuss evidence of the unit’s alignment with NGSS (see OR 10). Teachers analyzed and scored 
samples of student work and considered how the THSB unit is designed to achieve NGSS’s 
three-dimensional vision. For the study, teachers agreed to (a) use rubrics provided by the 
research team to score embedded assessment tasks involving written explanations for 15 of their 
students representing the range of performance in all of their classes; (b) summarize their 
findings across the sample, noting ideas understood by most students and ideas posing difficulty 
and what the implications might be for subsequent lessons; (c) email their reports to the research 
team along with scanned copies of the 15 students’ work within a day or two of when students 
completed each embedded assessment; and (d) provide feedback to students either by having the 
class evaluate sample explanations or by having students self-score and revise their own 
explanations. Figure 4 shows excerpts from the scoring table teachers used to record their 
judgments about the embedded assessment task that is reported on in this study (the actual 
worksheet is shown in OR 9). 


[insert Figure 4 here] 
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Teachers were told to spend about an hour on each embedded assessment task and were 
financially compensated for their time. By scoring each task immediately after students 
completed it, teachers would be able to use the results of their analyses to inform their 
instruction. The research team would use the results of the teachers’ analyses to measure their 
SMK and PCK. 

Indicators of Teacher Knowledge and Their Use in Judging Teacher Knowledge 

From the teacher reports on their students’ embedded assessments, we identified potential 
indicators of teachers’ SMK, PCK for science ideas and topics, and PCK for science inquiry that 
seemed to us to be most relevant to the NGSS science practice of constructing explanations (see 
Table 5). For example, would teachers notice if their students applied a science idea incorrectly, 
failed to cite evidence to support claims or had difficulty linking evidence to claims? Would 
teachers notice if their students were better at stating the science ideas than at using them to link 
evidence to claims? Would teachers notice if any students stated that models “proved” the 
claims, rather than merely giving them ideas about underlying molecular causes of the 
phenomena observed? Each component of teacher knowledge (except coherence) was evaluated 
in terms of a quantitative first indicator based on the percentage of times the teacher’s scores of 
student responses matched the researchers’ scores and a qualitative second indicator based on 
researchers’ analyses of the teacher’s written responses in their summaries (see p. 7 in OR 9). 
The indicators were designed to measure these aspects of teacher knowledge that seemed to us to 
be most relevant to NGSS. 

[insert Table 5 here] 
Knowledge of scientific ideas. People with a sophisticated understanding of science 


content are able to simplify the science ideas without distorting them. Therefore, we looked at 
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how well teachers were able to identify science ideas—often paraphrased—in their students’ 
responses to the embedded assessment tasks (Indicator 1) and how well teachers themselves 
paraphrased the ideas in their own summary reports (Indicator 2). 

Coherence. A coherent understanding includes knowing the connections among the 
science ideas and between the science ideas and lesson activities (Roseman, Stern, & Koppal, 
2010). After identifying difficulties exhibited in their students’ responses, teachers were asked to 
identify difficulties students had in the current lesson that could cause them problems in 
subsequent lessons or chapters (Indicator 3) or difficulties they carried over from earlier lessons 
that were continuing to cause them problems (Indicator 4). For example, students who were still 
confusing atoms and molecules at the end of Lesson 1.5 might have difficulty making sense of 
the chemical reaction involved in nylon formation during Lesson 1.6 and similar reactions 
involved in plant and animal growth in Chapters 3 and 4. If students were not yet using ideas 
about atom rearrangement to account for the formation of new substances, they could have 
problems explaining mass conservation and changes in measured mass in Chapters 2, 3, and 4. If 
teachers were able to foresee the implications of these problems for later lessons, it could 
indicate that they appreciate how ideas are connected to one another and to the unit’s activities. 

PCK: student ideas/misconceptions. Attending to students’ ideas starts with knowing 
relevant student misconceptions and how they might be manifest in students’ oral and written 
work. Teachers were asked whether their students’ responses included any of the misconceptions 
listed in the chart in Figure 4, along with any other misconceptions they could identify. 
Teachers’ knowledge of students’ ideas was measured by their ability to recognize 
misconceptions in the samples of their students’ work (Indicator 5) and their ability to 


characterize and give an example of the misconception or difficulty (Indicator 6). 
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PCK: science practices. Consistent with NGSS’s vision for three-dimensional learning, 
teachers’ knowledge of science practices was measured in the context of the specific phenomena 
students were asked to explain in the embedded assessment tasks and the science ideas they were 
expected to use in their explanations. Teachers’ knowledge of the practice of analyzing and 
interpreting data was judged by their ability to correctly score the evidence component of 
students’ explanations, which required teachers to decide whether or not the evidence students 
cited was relevant to and supported the claim (Indicator 7) and to comment in their summary 
reports on whether students did or did not cite evidence in their explanations (Indicator 8). 

Teachers’ knowledge of the practice of using models as a tool for reasoning about 
phenomena—-particularly those involving atoms and molecules—was judged by their ability to 
determine whether students’ use of models was consistent with the science ideas they cited in 
their explanations and whether students’ use of models helped to show that the claim in their 
explanation was reasonable (Indicator 9). Teachers also had to comment in their summary 
reports on whether their students did or did not use models appropriately in their explanations 
(Indicator 10). Lastly, teachers’ knowledge of the practice of using science ideas to link evidence 
to claims, another aspect of valid reasoning, required teachers to decide whether students applied 
the general principle in the science ideas to the specific claim and evidence (Indicator 11) and to 
comment in their summary reports on whether their students’ did or did not do so (Indicator 12). 
Scoring 

To arrive at a score for the quantitative first indicator of the teacher knowledge 
components (except for coherence) described above (Indicators 1, 5, 7, 9, and 11 in Table 4), 
researchers worked in pairs to independently score 30 sample student explanations, five from 


each teacher. Using the same rubric as the teachers (see Figure 4), the researchers scored each 


27 


students’ explanation as a 0, 1, or 2 based on whether the student (1) provided a correct answer, 
(2) stated the science idea, (3) used the science idea,(4) provided relevant evidence, and (5) used 
models in their explanation. The researchers also noted whether the student responses exhibited 
any of the likely misconceptions. The researchers then reconciled their judgments and compared 
their judgments to the teachers’ judgments. Agreement was reached when researcher and teacher 
judgments were within one point out of the two possible points for that component of the student 
answer. For misconceptions, teachers who called out a misconception directly on a student’s 
paper but failed to note it in the study table were given credit for having identified the 
misconception. Researcher and teacher judgments that differed by more than one point were 
considered to not match. For example, if the researcher did not think that a student used the 
science idea in her explanation (a score of 0), but the teacher though the student did use the 
science idea (a score of 2), this would not be considered a match. The percentage of times the 
teachers’ and researchers’ scores matched was calculated. Teachers received credit for the first 
indicator if their judgments matched the researcher judgments for at least 80% of student 
responses. 

To arrive at a score for the qualitative second indicator of the knowledge components 
described above, and for both of the coherence indicators (Indicators 3, 4, 6, 8, and 12 in Table 
4), researchers again worked in pairs and independently read the teachers’ responses to 
Questions 1-3 on p. 7 of OR 9 in their summary reports. The researchers identified examples that 
could count as evidence of teacher knowledge using the indicators in Table 5, and then 
reconciled any differences in judgments. Teachers received credit for the second indicator if they 


provided at least one example. 
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An overall rating was then assigned to each knowledge component. If the teacher met 
both indicators for a particular component, we concluded that there was robust evidence that the 
teacher had that knowledge. If only one indicator was met, we concluded that there was only 
some evidence that the teacher had the knowledge. If neither indicator was met, there was no 
evidence that the teacher had the knowledge. 

Results 
Table 6 presents scores for each teacher on each indicator of teacher knowledge. Table 7 
presents a few examples teachers provided in their summary reports that were counted as 
evidence for the qualitative second indicator. 


[Insert Tables 6 and 7 here] 
Subject Matter Knowledge 


On the Scientific Ideas indicators, all of the teachers’ judgments about their students’ 
science ideas matched researcher judgments for at least 80% of their students’ explanations 
(Indicator 1), and all of the teachers were able to paraphrase the ideas and cite evidence showing 
that their students understood the ideas (Indicator 2) (see Tables 6 and 7). Therefore, there was 
robust evidence that the teachers had an understanding of the science ideas. Moreover, Teachers 
B, C, and F reported on an alternative student explanation for why a chemical reaction does or 
does not take place when hydrogen peroxide is applied to a wound or when water is heated. 
Students justified the claim that a chemical reaction occurred in the first case because H2O and 
O2 are new substances and have different atomic compositions from H2O2 and that a chemical 
reaction did not occur in the second case because the ending substance H20 is the same as the 
starting substance. The three teachers scored their students’ responses consistent with this valid 
alternative approach and noted that they did so in their reports. On the Coherence indicators, four 


teachers provided some evidence of understanding implications of student difficulties for 
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subsequent lessons. For example, teachers B and D described difficulties their students could 
have in applying ideas about atom rearrangement (encountered in the context of small molecules 
in the current lesson) to the formation of polymers from monomers in subsequent lessons (see 
Table 7). Teacher C noted a difficulty her students still had with an idea from a previous lesson 
(the idea that substances are different because they have different arrangements of atoms) and 
described activities in subsequent lessons that might address the difficulty (see Table 7). None of 
the teachers provided more than a single example and two of the teachers provided no examples. 
PCK of Student Ideas/Misconceptions 

Two teachers provided robust evidence of understanding their students’ ideas, and four 
teachers provided some evidence (see Table 6). One teacher did not receive credit for the first 
indicator, but 70% of her judgments about students’ explanations agreed with researchers’ 
judgments. Three teachers who received credit for the second indicator provided specific 
evidence that their students confused atoms and molecules. As shown in Table 7, Teacher A also 
included an example of a student’s response indicating the student held the misconception that 
the production of a gas was always associated with a chemical reaction. Teacher F provided 
additional insight on the relationship between students’ lack of understanding of atom 
rearrangement (Science Idea #5) and difficulties distinguishing between the hydrogen peroxide 
and the heating water phenomena. Teacher F was one of the teachers who gave credit for her 
students’ alternative approach to explaining the hydrogen peroxide phenomenon but noted that 
those students were not using atom rearrangement in their explanations. 
PCK of Science Practices 

Most of the teachers’ responses showed some evidence of meeting the PCK of Science 


Practices indicators. Overall, the teachers scored highest on the Using Science Ideas in 
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Explanations indicators (see Table 6). For example, Teacher A noted that his students were able 
to use the science ideas in explanations (see Table 7), which was consistent with researcher 
judgments. Teachers E and B noted that their students had more difficulty using the science ideas 
than merely stating them. For the indicators on Using Evidence in Explanations, teachers’ reports 
provided less evidence that they were attending to their students’ use or lack of use of evidence 
in their explanations. While all teachers met the first indicator, only two teachers also met the 
second indicator. Teachers provided the least evidence for the Using Models in Explanations 
indicators. All but one met the first indicator, but only one teacher also met the second indicator. 
This indicates that the teachers’ responses to the summary report questions lacked an evaluation 
of their students’ strengths and difficulties with reasoning from models.Discussion 

Research Question 1 

Our adaptation of the Davis and Krajcik design heuristics can be used to focus curriculum 
developers’ thinking on the kinds of educative features that teachers will need to help them 
implement materials that support NGSS. Over time, as more curriculum materials are designed 
with NGSS in mind, it will be important to examine the nature of their educative features, the 
extent to which they are used, and their effectiveness in increasing teachers’ knowledge and 
skills with regard to NGSS. 

Given the magnitude of the changes called for in NGSS, the pace at which states and 
districts are adopting the new standards, and what research has to say about how teachers make 
use of educative materials, it is essential to prioritize what teachers most need to know and be 
able to do with regard to NGSS and what is practical within the context of a specific curriculum 
material. Our adaptation of the design heuristics and the educative features we developed for the 


THSB unit do not address every change called for in NGSS. No eight-week unit would be able to 
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accomplish that. Indeed, NGSS calls for seven conceptual shifts, each one involving a multitude 
of changes on the part of teachers, schools, curriculum and assessment developers, higher 
education, and more. Instead, we chose to focus on helping teachers understand at a deep level 
what it means to integrate NGSS disciplinary core ideas, crosscutting concepts, and science 
practices and to guide their students in drawing on these three dimensions to make sense of a 
specific set of phenomena. And as noted earlier, we decided to build most of the support for 
three-dimensional learning into the SE itself so that, for example, the scaffolding provided in the 
SE for helping students engage in the practice of explaining phenomena by drawing on evidence 
and science ideas also illustrated for teachers how the three dimensions can work together to 
promote understanding. Other developers will make different choices about which aspects of 
NGSS are most relevant to their materials, which aspects will require the most support, and how 
best to provide that support to teachers. Our use of the EQuIP Rubric criteria to analyze the 
THSB unit at various stages of development provided valuable insights on the extent to which 
the unit was achieving the goal of three-dimensional design. It is important to note, however, that 
understanding the criteria and applying them correctly and consistently is not a trivial 
undertaking. Much more clarification of the criteria as well as empirical results to support them 
will be needed in order for the EQuIP Rubric to have a significant impact on the design of 
materials. 

In developing the THSB unit, we were fortunate to work with a number of very talented, 
dedicated, and extremely busy teachers who tried out various iterations of student and teacher 
materials. We benefited from their feedback and from opportunities to observe how they used the 
materials in their classrooms. Our work confirms what Davis and Krajcik theorized: that “teacher 


learning will best be promoted by a set of complementary approaches, not by a single one” (p. 4, 
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2005). Some of the teachers in our study preferred to use the TE in the classroom while many 
others worked only with the SE. Some referred to the online resources regularly, while others did 
not. All, of course, participated in the face-to-face PD and used the student materials with 
fidelity. Determining the most effective mix of educative features will require much more 
research. In the meantime, providing multiple approaches to supporting teachers in implementing 
NGSS seems to make sense, particularly to address the specific needs of teachers at different 
grade levels. According to Banilower, Nelson, Trygstad, Smith, and Smith, (2013), for example, 
elementary teachers rarely have access to curriculum materials that are educative in any sense. 

Developers of new materials for NGSS have many challenges, chief among them is 
helping teachers understand and use the new standards. Our adaptation of the Davis and Krajcik 
design heuristics, our use of the EQuIP Rubric criteria, and the educative features developed for 
the THSB unit can serve as starting points for this effort. 
Research Question 2 

How teachers responded to the embedded assessment tasks provided considerable and 
different kinds of information on teachers’ SMK, PCK of student ideas/misconceptions, and 
PCK for science practices. The first indicator provided information about the extent to which 
teachers could use their knowledge to judge the different components of their students’ 
explanations. The second indicator provided information on teachers’ inclination to notice and 
comment on strengths and weaknesses of the various components of their students’ knowledge 
and explanation writing. Differences in teachers’ scores may also reflect differences in their 
writing ability and the lack of examples of what reports should look like. The time required for 
teachers to score their students’ explanations and then complete a written report may have been 


another factor: two hours rather than one hour per task is a more realistic expectation. 
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Furthermore, the teachers were teaching in different schools with different histories of 
implementing the Common Core English Language Arts standards, so teachers and their students 
varied in their experience with constructing explanations and arguments in the context of 
language arts. 

For the coherence component of SMK, teachers had less opportunity to receive a “robust” 
score because, unlike their scoring of the other components, they were not asked to quantitatively 
score students’ understanding of the storyline. Also, the second coherence indicator may have 
been less relevant for a task that occurs early in the unit. We might have elicited richer responses 
from teachers if we had asked them to look at the Content Storyline Map (see p. viii of OR 1) 
and describe ideas and lessons where problems they had observed with their students could have 
an impact later in the unit. Or we might have presented teachers with a hypothetical student 
difficulty and asked them to use the Content Storyline Map to identify lessons where the student 
might have problems. Providing a common example for teachers to address could also level the 
playing field for teachers whose own students exhibited few misconceptions. Given the success 
of THSB in reducing student misconceptions (Herrmann-Abell, Koppal, & Roseman, 2016), 
these strategies are worth considering. 

This study was designed to examine the usefulness of the educative features heuristics for 
designing materials for and tools to assess teacher learning, not to measure the effects of the 
educative features on teacher learning. However, data from previous work suggest that teachers 
involved in this study have learned a great deal from using the THSB unit. Using an online test 
to measure science content knowledge and knowledge of students’ ideas, Flanagan et al. (2013) 
reported that teachers’ knowledge of atoms/molecules and their inclination to use atom 


rearrangement and conservation to explain phenomena increased after PD and further increased 
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after using the unit. Teachers’ knowledge of students’ misconceptions, however, was minimal 
before PD and showed no increase. Difference between teacher knowledge reported in Flanagan 
et al. 2013 and in the study reported here may be due to: (1) considerably more support in the 
current iteration of the THSB for improving teachers’ knowledge and practice, leading to higher 
quality student explanations (Herrmann Abell & Roseman, 2016); (2) wider adoption of NGSS 
leading to an increase in teachers’ attention to science practices, especially data analysis and 
explanation; and (3) most teachers’ knowledge increasing as their experience with the THSB unit 
increased. 

The research literature has pointed to the difficulty of developing measures of teacher 
knowledge that are both valid and authentic. What is more, our review of the literature found no 
instruments that were designed to measure NGSS-related teacher knowledge. The measure 
developed for this study, therefore, may be the first to be applied to a curriculum material that is 
designed explicitly to align with NGSS and the first to operationalize teacher knowledge of the 
NGSS vision of three-dimensional learning in an authentic way, that is, in the context of 
teachers’ analysis and synthesis of their students’ written responses to embedded assessments. 
The high level of compliance, which is not always the case with more traditional measures, 
might be because the teachers did not perceive the measure as a “test” of their knowledge but 
rather as a task that would provide useful insights into their students’ thinking. Our study, unlike 
others, focused specifically on science and on a curriculum material rather than on video (see, for 
example, Kerstin, 2008) and on in-service teachers with experience in using the THSB unit 
rather than on pre-service teachers (see, for example, Schwarz et al., 2008). 

While more work is needed to refine and validate the measure, even in its preliminary 


form it can shed light on teacher knowledge. Further studies are also needed to determine 
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whether the teacher knowledge we observed resulted from their use of the educative features (or 
components of those features) in the THSB unit. The existence of validated measures based on 
the one described here would make such studies possible. 
Conclusions 
We have demonstrated that the heuristics proposed by Davis and Krajcik (2005) for the 
design of educative curriculum materials can be adapted for use with materials that aim to 
achieve the NGSS vision of three-dimensional science education. Analysis of drafts of the unit 
using Project 2061’s research-based textbook evaluation criteria (AAAS, 2005) and EQuIP’s 
adaptation of them (Achieve, 2016) provided additional guidance in what the final product 
should look like (Roseman, Fortus, & Reiser, 2015; Roseman, Herrmann Abell, & Kruse, 2016). 
Our findings from the use of the embedded assessments suggest that the same design heuristics 
can also guide the development of measures to evaluate teacher knowledge and practice. 
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Table 1 


Instructional Phenomena in the THSB Unit 


Conceptual Focus 


What Students Experience 


1: New substances form 
during chemical reactions 
because atoms rearrange 
to form new molecules 


Observing, modeling, and explaining how different ending substances 
can form from starting substances when: 

e Vinegar is mixed with baking soda 

e Iron is exposed to air 

e Hexamethylenediamine is mixed with adipic acid 


2: Mass is conserved in 
chemical reactions because 
atoms are conserved 


Observing, modeling and explaining how the measured mass of a 
system can change even though atoms aren’t created or destroyed 
(same reactions used in Chapter 1) 


3: Plant growth involves 
chemical reactions in 
which atoms rearrange and 
are conserved 


Observing, modeling, and explaining how plants produce 
carbohydrates for growth from substances in their environment and 
increase in mass without violating conservation principles when: 

e Plants produce “CeH120¢ from “CO2 

e Plants produce "802 not C6H12*8O¢ from H2!%O 

e Cress make more “4C-cellulose without herbicide than with it 


4: Animal growth involves 
chemical reactions in 
which atoms rearrange and 
are conserved 


Observing, modeling and explaining how animals produce proteins 
for growth that are different from what they eat and increase in mass 
without violating conservation principles when: 
e Asnake eats only eggs but can its replace shed skin 
e Humans eat muscles but can make tendons 
e Herring eat *4C-labeled brine shrimp and make “C-labeled 
body structures 


Table 2 


Embedded Assessments in the THSB Unit 


Chapter, Focus 


Embedded Assessment 


1, Atom rearrangement explains the 
production of new substances during 
chemical reactions 


a. Hydrogen peroxide is a clear, colorless liquid. When 
you apply it to a cut or scrape on your skin, you will see 
bubbles and you may hear a “‘fizzing” sound. Figure 1.3 
uses LEGO® models to represent hydrogen peroxide on 
the left. On the right are models showing what happens to 
hydrogen peroxide when you apply it to a cut or scrape on 
your skin. 

Do you think a chemical reaction occurs when hydrogen 
peroxide is put on a wound? Explain why you do or do not 
think so in terms of atoms and molecules. Your 
explanation should use science ideas, evidence, and 
models to support your claim. (Use the table for your 
notes) 

b. As a pot of water is heated on a stove, bubbles of gas 
form in the liquid. Figure 1.4 uses LEGO models to 
represent the molecules of water in the pot on the left. On 
the right are models of the molecules inside the bubbles 
that form. 

Do you think a chemical reaction occurs when water is 
heated on a stove? Explain why you do or do not think so, 
using science ideas, evidence, and models to support your 
claim. (Use the table for your notes) 


2, Atom conservation explains mass 
conservation and changes in measured mass 
during chemical reactions in open systems 


The Statue of Liberty is made up of copper (Cu atoms). 
But the statue doesn’t have the shiny, orange-brown 
appearance of copper. Instead, it is green. Why? After 
being exposed for many years to oxygen, carbon dioxide, 
and water vapor in the air, a thin layer of green copper 
carbonate (CuCO3 molecules) formed on the copper statue. 
Answer the following questions and be sure to use ideas 
about atoms in your explanation, and to write an 
explanation that meets the Explanation Quality Criteria. 
a. Is the change a chemical reaction? Explain. 
b. Do you think the Statue of Liberty has less mass, 
the same mass, or more mass now than when it 
was first made? Explain. 


3, Atom rearrangement and conservation 
explains the production of new substances 
for building plant body structures 


The paper on the next page describes an experiment 
carried out by a scientist named J. B. Van Helmont that 
was published nearly four centuries ago (1662). 

a. Read the paper, underline Dr. Van Helmont’s 
conclusion, and circle his summary of his 
evidence. 

b. Does Van Helmont’s evidence support his 
conclusion? Why or why not? Does his 
explanation meet the elements of the Explanation 
Quality Criteria? 


c. Write a better explanation for where most of the 
mass of a dry willow tree comes from. 


4, Atom rearrangement and conservation When you get a cut on your hand, your body builds scar 
explains the production of new substances tissue to seal up the cut. Scar tissue is made up of the 
for building animal body structures protein collagen. Explain how this can happen. Be sure 


your explanation meets the Explanation Quality Criteria. 


Table 3 


Study Participants 
Teacher Characteristics Sample 
Race 
Black 1 teacher 
White 5 teachers 
Gender 
Female 5 teachers 
Male 1 teacher 
Avg. Yrs. Teaching Experience 
Physical Science 9.3 years 
Life Science 7.4 years 
Education 
Undergraduate degrees 5 (biology) 


Graduate degrees 


1 (education) 
1 (education) 


Experience with THSB unit 
One Year 
Two Years 
Three Years 


2 teachers 
2 teachers 
2 teachers 


Table 4 


Indictors of Teacher Knowledge 


Indicators of Teacher Knowledge 


Subject Matter Knowledge 


Scientific Ideas 


1. 


Teacher correctly identifies correct/incorrect science ideas in their 
students’ responses (based on how well teacher scores match 
researcher scores on elements involving stating science ideas) 


Teacher’s summary correctly paraphrases the science idea (note which 
science ideas are correctly paraphrased) and gives evidence-based 
example of students’ correctly using the science idea 


Coherence 


Teacher’s summary notes a specific idea/activity beyond the current 
lesson where identified students’ difficulties with ideas could cause 
problems (mention of specific practice counts as evidence for PCK for 
that practice) 


4. 


Teacher’s summary notes a specific idea/activity from an earlier lesson 
that was needed but not learned well enough for students to build on 


Pedagogical Content Knowledge 


Student ideas/ 
misconceptions 


5. Teacher correctly identifies examples of misconceptions in their 
students’ responses (based on match of teacher judgments to 
researcher judgments) 

6. Teacher’s summary gives evidence-based example(s) of student 


misconceptions or difficulties with ideas in summary report (e.g., 
confusing atoms/molecules, thinking the mass of the willow tree comes 
only from water, overestimating the contribution of water to the 
increase in the willow tree’s mass, confusion between total mass and 
measured mass in terms of what is conserved) 


Pedagogical Content Knowledge for Science Practices 


7. Teacher correctly scores the evidence component of students’ 
Evidence explanations 
8. Teacher’s summary mentions citing evidence as a strength or difficulty 
in student explanations 
9. Teacher correctly scores the models component of students’ 
Models explanations 
10. Teacher’s summary mentions reasoning from models (could be 
mentioned when describing student difficulties) 
11. Teacher correctly scores the using science ideas component of 
Using Science students’ explanations 
Ideas 12. Teacher’s summary mentions using science ideas (could be mentioned 


when describing student difficulties) 


Table 5 


Scores on Indicators of Teacher Knowledge for THSB Teachers 


Teacher A B C D E F 
Years using THSB 2 3 1 1 2 3 
Subject Matter Knowledge 
Scientific Ideas Robust Robust Robust Robus Robust Robust 
* * t * 
1. Teacher correctly identifies correct/incorrect the statement of 1 1 1 1 1 1 
science ideas in their students’ responses (100%) (90%) (80%) (100% (90%) (100%) 
) 
2. Teacher’s summary correctly paraphrases the science idea and 1 1 1 1 1 1 
gives evidence-based examples of students correctly using the idea 
(note which science ideas are correctly paraphrased) 
Coherence None Some Some Some _ None Some 
1. Teacher's summary notes a specific idea/activity beyond the current 0 1 1 1 0 1 
lesson where identified students’ difficulties with ideas could cause 
problems 
2. Teacher's summary notes a specific idea/activity from an earlier 0 0 0 0 0 0 
lesson that was needed but not learned well enough for students to 
build on 
Pedagogical Content Knowledge: Student ideas/misconceptions Robust Some Some Some Some_ Robust 
Teacher correctly identifies examples of misconceptions in their 1 1 0 1 1 1 
students’ responses (90%) (80%) (70%) (80%) (80%) (100%) 
2. Teacher’s summary gives evidence-based example(s) of student 1 0 1 0 0 1 
misconceptions or difficulties with ideas in summary report (e.g., 
confusing atoms/molecules, thinking a substance’s properties can 
change) 
Pedagogical Content Knowledge for Science Practices: Explanations 
Using Evidence Some Some Some’ Robus Robust Some 
t 
1. Teacher correctly scores the evidence component of students’ 1 1 1 1 1 1 
explanations (100%) (90%) (90%) (90%) (100%) (100%) 
2. Teacher's summary mentions citing evidence as a strength or 0 0 0 1 1 0 
difficulty in student explanations 
Using Models Some Some None Robus Some Some 
t 
1. Teacher correctly scores the models component of students’ 1 1 0 1 1 1 


explanations (100%) (90%) (50%) (80%) (100%) (100%) 
0 1 0 0 


2. Teacher's summary mentions reasoning from models as a strength 0 0 
or difficulty in student explanations 
Using Scientific Ideas Robust Robust Robust None Robust Some 
1. Teacher correctly scores the using science ideas component of 1 1 1 0 1 1 
students’ explanations (100%) (90%) (80%) (70%) (100%) (100%) 
2. Teacher's summary mentions using science ideas as a strength or 1 1 1 0 1 0 


difficulty in student explanations 


Total score (out of 12) 8 8 7 8 8 8 


“Teacher notes students have cited/used alternative science ideas (#2 & 4 versus #5). 


Table 6 


Evidence from Teachers’ Reports on Students’ Explanations 


Type of Teacher: Example Provided 
Knowledge 
Teacher E: Almost all students understand Science Idea #5, that when the atoms in a 

S molecule rearrange that a chemical reaction has occurred. | say this because almost all 
3 students correctly identified the chemical vs. non-chemical reactions. They also stated 
= SI#5 in their answers. For example, student #1315610004 says that the boiling water is 
€ not a chemical reaction “because all of the molecules are the same” and student 

3 #1315610022 stated that, “for the substance to change it needs to have the atoms 


rearrange.” 


Subject Matter Knowledge 


Teacher B: My students will most likely have difficulty in understanding the chemical 
reaction that occurs between monomers to create polymers. | will need to walk them 
through the formation of water so they can understand that something new is made. 
They will think that the polymer is not something new because only the atoms at the 
end come off. 


ss) Teacher D: about % of my students are having a hard time conceptualizing monomers 
o and polymers” and that “Science Idea #6 [which applies the idea that atom 
2 rearrangement is why new substances are produced during chemical reactions to the 
8 specific case of polymer formation] will be intimidating to them, but as we move 
forward | see it being a building block for better understanding [of plant and animal 
growth]. 
Teacher C: Science Idea #4 about the type, number and arrangement being a different 
substance seems to be weakly understood at this point. With Chapter 2 and the 
counting of the number and type of atoms this should be strongly enhanced. 
Confusing atoms and molecules: 
Teacher A: “The original substance, hydrogen peroxide, had a chemical formula of 
H2O>2. After the reaction, the molecules have been separated, as the new chemical 
& formulas are O and H20.” (#1315521054) 
2 “Chemical reactions have taken place if the molecules separate (science idea #5). The 
3 molecules did not separate, as shown in the model, so a chemical reaction didn’t take 
2/2 place.” (#1315521054) 
€ s A gas always indicates a chemical reaction has occurred: 
€ € Teacher A: “Also, when touching the wound, the hydrogen peroxide starts to bubble 
9/3 and fizz which indicates a chemical reaction.” (#1315521001) 
Ss a Difficulty understanding atom rearrangement: 
o Teacher F: For the students who are struggling with Science Idea #5, they are also the 
co ones that are unsure of what exactly is happening during the reaction. One of the 
ne students, in their response, seemed to think that there was an increase, or 


“replication,” of atoms the H202 question. This was also the student who thought that 
water heating did represent a chemical reaction because of the change in state, from 
liquid to gas. 


PCK for Science Practices: Explanation 


Teacher E: the difficulty | see some of my students having is connecting the Science 


tO) 

e b. Ideas to the explanations. 

8 © | Teacher A: the majority of students accurately paraphrased Idea #5 and used it to link 

we © | evidence to answer. 

2 Teacher B: 8/15 students were able to use the idea [#5] to help explain their answer. 
Teacher E: The idea that might be causing problems is SI#2 [substances with different 

® | properties form during chemical reactions]. Several students listed this idea as 

= 3 evidence. This is a problem because the properties of the substances were never 

=) > talked about in the prompts. Students just used prior knowledge, rather than the text, 


to answer the question. 


Using 
Models 


Teacher D: determining the difference between evidence and models is particularly 
challenging.” 


#14. When plants grow or repair, they 
increase in mass. Atoms are conserved 
when plants grow: The increase in measured 
mass comes from the incorporation of 
atoms from molecules that were originally 
outside of the plants’ bodies. (from LS1.C 
and CC5) THSB Lessons 1.1, 3.1, 3.4, 3.5, 4.5 


#13. To build body structures for 
growth and repair, plants use glucose 
monomers to make carbohydrate 
polymers and water molecules. Atoms 
are rearranged during this chemical 
reaction. (from LS1.C) 

'THSB Lesson 3.4 


#12. Plants use carbon 
dioxide and water 
molecules in their 
environment to make 
glucose and oxygen 
molecules. Atoms are 
rearranged during this 
chemical reaction. 
(from LS1.C) 

THSB Lesson 3.3 


#11 Carbohydrates are the 
main polymers making up 
plant body structures. 
Carbohydrate polymers are 
molecules made of glucose 
monomers. Different 
carbohydrate polymers have 
different properties because 
they are made of different 
numbers and arrangements 
of glucose monomers. 

(from LS1.C) 

THSB Lessons 3.1, 3.2 


plant growth 


Figure 1. The content storyline map displays the sequence and interconnections of ideas targeted in the THSB unit. 


Toward High School Biology (THSB) 


Year 6 Content Storyline 


#10. The measured mass of reactants and products is not always the 
same as the total mass. The measured mass changes if substances 
(often gases) enter or leave the system. This is because atoms that 
make up these substances enter or leave the system. (from PS1.B 
and CC5) THSB Lessons 2.1, 2.3 


#9. Because the mass of a particular atom does not change and 
because the number of each type of atom does not change, the} 
total mass of the matter does not change during a chemical 
reaction even though atoms are rearranged. (from PS1.B) 

THSB Lesson 2.2 


#7. Atoms are not created or 
destroyed during chemical 
reactions, so the total number of 
leach type of atom remains the 
same (atoms are conserved). 
(from PS1.B) 

|THSB Lesson 2.2 


#8. The mass of a particular atom 
does not change during a chemical 
reaction, so a given number of that 
type of atom will always have the 
same total mass. (from PS1.B) 
THSB Lesson 2.2 


#6. Very large molecules called polymers 
can be formed by reacting small molecules 
(monomers) together. Because monomers 
can react in two places, it is possible for 
each monomer to react with two other 
monomers to form long polymer chains. 
Each time a monomer is added to the chain, 
atoms are rearranged and another 
molecule, typically water, is formed. 

(from PS1.B) 

THSB Lessson 1.6 


#5. During chemical reactions, atoms that make up molecules of the starting 
substances (called reactants) separate from one another and connect in 
different ways to form the molecules of the ending substances (called 
products). Because the arrangementof atoms in the products is different 
from the arrangement of atoms in the reactants, the products of a chemical 
reaction have different properties from the reactants. (from PS1.B) 

THSB Lesson 1.5 


#4. Each substance is made up of a single type of 
atom or molecule. The properties of a substance are 
determined by the type, number, and arrangement 
of atoms that the substance is made up of. Because 
no two substances are made up of the same 
arrangement of atoms, no two substances have the 
same set of properties. (from PS1.A) 

THSB Lesson 1.4 


#2. Changes during which new 
substances form are called chemical 
reactions. The correlation of increasing 
lamounts of ending substances with 
decreasing amounts of the starting 
substances provides evidence that the 
new substances result from an 
interaction between the starting 
substances. (from PS1.B) 

ITHSB Lessons 1.2, 1.3 


#3. A molecule is made up of 
two or more atoms connected 
together in a specific 
arrangement. (from PS1.A) 
THSB Lesson 1.4 


#1. Every substance has a unique set of characteristic properties, such as color, odor, density, 
melting point, conductivity, solubility, and how it behaves (such as in limewater and glowing 
splint tests). The properties of substances can be observed or measured and used to decide 
if two substances are the same or different. (from PS1.A) 

THSB Lesson 1.2 


chemistry 
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#17. When animals grow or repair, they 
increase in mass. Atoms are conserved 
when animals grow: The increase in 
measured mass comes from the 
incorporation of atoms from molecules 
that were originally outside of the 
animals’ bodies. (from LS1.C and CC5) 
THSB Lessons 1.1, 4.1, 4.3, 4.4, 4.5 


#16. The process by which proteins from 
food become part of animals’ body 
structures involves chemical reactions in 
which the proteins from food are broken 
down into amino acid monomers, and these 
monomers are used to build the protein 
polymers that make up their body structures. 
Atoms are rearranged during both the 
breakdown and building of protein 
polymers. (from LS1.C) 

THSB Lessons 4.3 


#15. Proteins are the main polymers making up 
animal body structures. Protein polymers are 
molecules made of amino acid monomers. Different 
protein polymers have different properties because 
they are made up of different types, numbers, and 
arrangements of amino acid monomers. (from 
LS1.C) 

THSB Lessons 4.1, 4.2 


animal growth 


Steel Wool (Iron) and Air 


(Substances we start with in the container) 


Fe 
(iron) O2 


(oxygen gas) 
Fe 
(ron) a 
Se Bw: 
(irom eis] focen gas) 


Fe O, 
(iron) (oxygen gas) 
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Steel Wool (Iron) and Air 


(Substances we end up with in the container) 


Fe,O, 
(iron oxide/rust) 


Fe,O0, 
(iron oxide/rust) 


Take apart starting substances from the other side 
and build the substances above with the same LEGO® bricks. 
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Baking Soda and Vinegar 


(Substances we start with in the container) 


HC,H.,O, 
(acetic acid/ 
vinegar) 


NaHCO, 
(Sodium bicarbonate/ 
baking soda) 
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Baking Soda and Vinegar 


(Substances we end up with in the container) 


(sodium acetate 


CO, 
(carbon dioxide) 


Take apart starting substances from the other side 
and build the substances above with the same LEGO® bricks. 
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Figure 4. Scoring rubrics for Chapter 1 embedded assessment task. Teachers scored each students’ explanation, assigning up to 2 points for each 
element and noting if the explanation reflected any indicators of difficulty. 

Question 1a: Hydrogen peroxide is a clear, colorless liquid. When you apply it to a cut or scrape on your skin, you will see bubbles and you 

may hear a “fizzing” sound. Figure 1.3 uses LEGO models to represent hydrogen peroxide on the left. On the right are models showing what 
happens to hydrogen peroxide when you apply it to a cut or scrape on your skin. Do you think a chemical reaction occurs when hydrogen 
peroxide is put on a wound? Explain your answer, using evidence, science ideas, and models as needed to support your answer (use the 

table for your notes). 


Student ID Codes: 
Elements of an Ideal Response 


Claim A chemical reaction occurs. 
States If atoms rearrange to form new molecules then a chemical reaction occurs 
Science | (Science Idea #5). 
Idea 


Uses The LEGO models in Figure 6.1 show that H,O and O, have different 
Evidence| arrangements of atoms (are different molecules) from HO). 


Uses The LEGO models show that H,O and O, molecules could have been formed 
Models | from rearranging the atoms of H,O, during a chemical reaction. 

Uses O and H atoms must disconnect and join in different ways to form H,O and 
Science | O, from H,0>. 

Idea 


Misconceptions 
The ending substances are made up of the same molecules as starting substances. 
(confusing atoms and molecules) 
Substances change their properties during chemical changes, but their molecules stay 
the same. 
The atoms and molecules of the starting substances are completely independent 
of/unrelated to the ending substances. 
Gas has formed; gas formation always indicates a chemical reaction has occurred. 
The number of models increased, so atoms cannot have just rearranged. (confusing 
atoms and molecules, difficulties with conservation of matter) 


